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The purpose of this..publicatton, is to revie 
the VocationajL Evaluation and l^prk Adjtistment Assbciation s 
testing standards a^s ifell as to provide insight and underst 
into the underlying'purposes of the guidelines. The first o 
sections focuses on^those sta^ndai-us designed to insure that 
saaple ■actuali»y measures what* it 'purports, to measure and is 
representative of real competitive worker skills* Secl^ion 2 
the^ requirements which define wl^t constitutes work sample 
standardisation. Specific reguirements discussed are joS an 
relation shit);.' prerequisites; work sao^jle purpose;' matexials 
eqaipment used; test preparation and layout; ^instructions^ 
error, and scoring instructions: and instructions for in^er 
scores. The final section expound-^ upon the standard which 
work samples to have appi^opriate competitive ot indus-t^rial 
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AN INTERPRETATION OF VEWAAACARF 

. . WORK SAMPLE STANDARDS 

< •. * 

' ' . Introduction ' ' 

Of the four major Vocational evaluation teqtiniques--work samples, psycho- 
logical testing, situational assessment, and job site: evaluation--work sample 
testing has become one of the most often used methods:. Indeed, in many re- 
spects it is a process that distinguishes the field of vocational evaluation 
from other more traditional psychological testtrig approaches. Yet with the 
increased application of this ^assessment technique, a corresponding* responsi- 
bility to assure that if is carried on in a professional manner .has arisen. 
In an effort to ^neet this charge, the Vocational Evaluation and Work Adjustment 
Association (VEWAA) has developed minimum standards to be applied to woric sam- . 
files. .Tftese standards have been adopted by the Commission on Accreditation of 
Rehabilitation Facilities (CARF) because they provide a sound basis for the 
development of reliable work samples. • T 

..''The present CARF work sampl^guidel ines raad as follows: 

3.4.3.1,1.7,2 If, work samples are used: 

* ' a. the vocational evaluation service work samples / 

resources- shall- be 'representative of realistic 
competitive worker skills. 

b. work samples shall be establ ished'by an analysis 
• of job .tasks or traits related to a specific^ 

area of work, and be standardized as to materials, 
layout, instructions, and scoring. 

c. competitive, norms or industrial standards shall 
be establ i shed ^and used., (p. 28) 

Thus/ it is clear that the guidelines fqr some of the content, the struc- 
^turej and the developm^t of work samples are relatively, specific. The purpose* 
of tMs publication is to review each of tjie'work sample testing 'standards as 
well as provide insight and understanding into th^underlying purposes of the 
guidelines. With this infDrmation in mind, vdcatiqnal evaluators and program 
administrators should be able to better assess the effectiveness and qual-ity of 
their wolrk samples with regard to: (1) meeting the needs of ^their clients and 
referral sources -in terms of using valid and reliable assessment techniques, ' 
and (2) satisfying prx>fessft>nal standards as well a© CARF accreditation stan- ^ 
dards for vocational evaluation programs, 'in addition, the guidelines should 
sferve as a valuable reference for selecting work samples Which satisfy these - 
reqCiii^ements and, therefore, are most .likely to provide reliable and valid 
information as to client capabilities and limitations. 
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Standard 3.4.3.1.1.7.2a of the CARF Standards Manual for Rehabilitation r. 
Facil ities indicates that if work samples are used, "the vocational evaluation 
service work samples resources shall be representative of realistic competitive 
worker skills" (p. 28). A joint CARF-VEWAA interpretation of- tfiis standard 
goes on to explain:" ' . * 

Evaluation programs using work samples that purport to be rep- 
resentative of specific work skills should be able to provide 
documefttation that they olS'. Documentation might include the ^ 
considered judgement of appropriate employers or. employees as 
to the ability of a work sample to measure skills related to a 
particular occuphtion. - (Vocational Evaluation a^d VTorR "AT- " 
justmqnt Association and Commission- on Accreditation of* Reha- 
bilitation Facilities, 1978, p. 6) 

■ . • ■ " . * ' 

The major implication of this standard and the interpreHation is that 
work samples should have evidence of validity. There must^be documentation, 
that a g^^/en work sample really measures* wh«t it purports to measure. In other 
words., if a work sample says that itapeasures a client's ability to work as a 
mail sorter, there should be evidence that it actually does so. In essence, 
this requires that the work sample contain the critical elements necessary 
for satisfactory mail sorting performance, and that |;hey can -be reliably 
measured and analyzed. Althougli knowledge of how to establish a test's 
validity is ce-rtainly important, it is mot the purpose of th.is publication 
to discuss -the various measures of validity and \\m they are related to work 
sampjes. Interested* readers ar.e referred to Dunn (r971) for a brief review . 
of this subject. ' ^ 

- " - ' * 

The phrase "rejjresentative of.realistic competitive worker skills" is of • 
criticar importance to understanding -this guidel in^. It refers to the idea 
that a work sample should measure the critical elements, e.^g. , aptitudes, * 
traits-, and skills-, actually required to perform a jobTound in the labor 
market. It fhould not measure elements that are not required for satisfactory,, 
performance, gather it sho-uld include elements which ar'e necessary- 'for com- 
petitive performartqe. For example', if a work sample contains all the elements 
required to perform the job of lathe operator, one could certainly saythat it. 
represents real competitive worker skills. On the other hand, i-f th& same work 
sample required the client, to perform a task whi(?h was* not piirt of the actu51 
jot>, e.g., memorize an operator's manual, one shduld recognize that the work / 
scampi e is no longer representative of real work- requirements, becaig^e the' 
ability to Ihemorize a manual is not actually required on. the job. '^Similarly, 
if a critical aspect of tKe job is the ability to read blueprints, and this 
skill is not included in the work Sample, .'then it is also not *ully ^repre^en- 
t'ativ'e of: all the real requirements of the "job. ' » 



. Therefore, if a work sample purports to measure one's aijility to work in / 
a fiveQ job or job area, it must'tontain the'crucial elements Required for 
\ satisfactory job performance. Likewise, if a work sample claifhs to, measure • ' 
/i"- given t^jiit, that trait should be. clearly defined and there should be eyi- 
*dence .that the trait has been accurately incorporated ill the work sample Jnd 
is ^^tematically measured. Jf th^ individual traits are described as being * 
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critical for satisf<actory job performance, thet'e must b^" documentation that it , 
is actually true.' Such' eviidence may be obtained from job analysis, -task 
analysis* Dictionary of Occupational Titles (DOT) data,. etc. ' ^ . 



The standard also.indicates that-a'work sample should ^e^r^aliitic.. 
Basically this means that it should re'late to -actual work. Thus, "when a client 
is performing a work, sample, he can see that the work he is doing is similar 
to competitive wor-k which really exists in industry. This lends credibilHy 
to the assessment instrument in that it is no longer an abstract concept or 
•process that lacks me'aningfulness fj^r the client.. This is qjie. Of the advan- 
tages work samples have over paper and pencil tests which clients often 
. '"^mistakenly view as- Jack i fig- any relationship- to real work: ... • '^^ 

* * * • ■■ . ' • ' • ■ 

Once it is recognized that work samples should contain th4 -elements that 

are. crucial to satisfactory j'ob performance, as well as be realistic, it fol- 
lows that one^ must ask how these elements cat^^ be objectively identified. The 
^ "1-irst halt of part "ft" or the standard, point^ gut t^e ^1fletha|^, fxrr identi P yinq 
.,. realistic competitive worker skills. It rfeads: "wprk samp.les shall be estab- 

|_ » lished by an analysis, of job tasks or 'traits -related to a specific area of 
. ' / , wor]^" (p. 28). Job analysis is probably the ms< effective and efficient waV 
. ' "to determine the critical aspects of a job. It essentially cQnsists-.of a/ia- ; 
lyzing the specific job related tasks, aptitudes, physical demands, machirles, 
tools, equipment, interests, etc., required foy\ satisfactory job performance-. 
Once this information is available, the evaluator may, begin to* incorporate 
' these elements .into the work sample. Cefr.tainly the evaluator should a.ttempt 
to include as many of the job tasks jn the work'^^nple as is practically 
.feasible. If the work sample* entirely replicates a job performed in indust.ry 
in terms of equipment, tools, production methods , work standards, etc^, it is • 
better 'thought of as a job sample. ' The t'EWAA-CARF glossary defines job 
sample as: .• * . ' 

Job Sample - Those worksamples that in thei^ entirety are replicated 

* directly f^om industry and include the- equipment, tools, raw mate- 
rials, 'exact procedur^es, and work standards of the job. (Vocational 

• Evaluation ^^nd Work Adjustment Association and. Commission on Accred- 
itation of^ Rehabil itatipn Facili't^es^ ^78, p. 20) ' ' 

'a job sample can be very useful* in assessing a clienfs potential to do a . 
specif ic^ job, but it still has limitatiohs; the primary one being that the wofk 
environment may not be entirely duplicated, ftis is .a difficult and often 
impossible task, since there is such a large namber of uncontrol;led variables 
'\ that make up any given work setting. But to the maximum extent possible^ work 
/ and job samples should 'simulate the work environment along with'^the work tasks 

/ , as closely as possible. In any case, it is imperative that the critical job 

"elements be included in the work sample. Otherwise, a client, may be ^ble to 
perform well on a wqrk sample, but cannot actually perform the job. For ex- 
' -ample, although a. bricklayer may-onfy spend a small portion of his time 
• jneasuring- a building layout, this skill must be included in the -work sample, ^ 
because without it th^ client will 'not be able to work competitively. Thus,*" 

— — t-he-t-erm critical refers ,to those elemefits of a job which are essential to 

^ satisfactory overall job performance. An -example of a noncritic'al element 
which need not be incorporated in a bricklayer work sample is fine f-inger 
. , dexterity. Although a bricklayer might have to periodically manipulate small 
objects, it generally fed not 'be done at exceptionally rapid rates such as 
are common to ,aSsembly type operations. The evaluator is. primarily concerned 

• * • . * * 

■ • .■ . . . • . ■ • 

/ • 
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with maki^ig sure tliat- the client simply 4an manipulate small objects. A 
client's abjlity to do such a task can generally be intuitively assessed 
*an evalqator. • " ' \ ' ' >^** 

^ ' I- ^ .. . 

Since it is- often unreal i stic to "expect ^~ work SMil5T^"^tD~c0nt2riTi^tr-tfie-v 
elements of a job, it is important that somewhere in the manual the elements 
which, were ^t included in the work sample be listed. In'this way the evalira- 
tor is made/ aware . of the limitations of the work sample and in many cases, he 
will be able to utilize different cqmbinations of work s^amples to supplement 
each other. For example, if a job such as photppopy machine operc^or Requires 
tne worker to operate a cash register 5% of the time, it may not be feasible 
to include an expensive cash register" as part of the work sample. Yet once 
tHis problem is pointed out, many evaluation arograms which already possess 
cashiering assessment work samples' will be able to supplement the photocopy 
work sample in such a way that a more reliable and thorough' assessment of 
Jjje client's abilities is possible. \^ 

Wofk "Samples shoul'd'not only have the" critical elements of a job, but 
they sh'ould also contain those elements in^a proportion similar to that of 
the job. Thus,, if a telephone solicitor spends 90% of his time talking on 
the 'phone and 10% of the remaining time recording information, the work 
sample* should reflect the proportional relationship of the elements. There- 
fore, a 'telephone solicitor work sample should not have a client spend 30% 
of his time on the phone and 70% of the time record.ing information.. Instead. • 
the 90/10% relationship would be incorporated into tlie work sample so as to ' 

provi(ie a more real istic simulation of the job^ ' ' 

... • 

Other techniques beside,s job analysis are* also available for determining 
job,e]ements. Task analysis is^tne method. ^It basically involves analysing 
in a step-by-step manner^ the procedu^'es involved in the individual tasks that 
make up a job.. In other cases, evaluators may be able to obtain detailed jo^ , 

?escriptions which specify the traits, aptitudes,- etc. , required as well as 
he amount of time employees are expected to spend- on gpeoific Job., tasks for 
•satisfactory performance. This information may be available from the per- 
sonnel department of public and private businesses, state employment service 
offices^ or occupatijDnal analysis field laboratories of the Department of 
Labor. Another rnetfTod is .to analyze the job descr4ptions, worker traits. 
Worker functions, etc! , "provided in the DOT."^ Howe>ver, the major weakness with 
tk\\s approach is that the DOT information' may differ from the individual job 
requirements ot local' industry. In any case, one of t^e ntost (mportant reasons 
for analyzing a job should be t,o accuratel^;determine thevelements that are 
important to a job' so that they may be incorporated in a work sample. As 
mentioned previously,' this process not oi^ly helps insure »the representativeness 
of real istic competitive worker skills , but al so .[Drovides documented evidence 
<that an analysis of job requirements has taken pla(fe. 

Although the importance of^insuring re^resentat:iveness aS well as how this 
goal can, be achieved thas |been pointed oiit, there- has ^been no discussion as to 
the p'roblem of how many or how much of the actual job elements must be "included 
in a work sample.. .For. example, once an analysis of the job has been completed 
and? as many of the elements as possible have been incorporated into the work 
sample, the evaluator can compare the degree to which the work* sample contains 
the elements of the job. Look at the following fcxamplge: 



Tel ephone So.l i c i tor , 



DOT 

I 



Code 299.357-014 



Oob Analysis \Descriptjofi of Tasks 



Calls prospective customers ot) phone, 
explains merchand^'se, and solicits 
purchases. (70%>)\ 

Records names and addresses of pyr- 
chasers. as well as \amount of 
purchase. (15%) ' ^ 

1 ■■ ■ ■ '\ • 
Reads| telephone book, to obtain 
phone numbers of prospective * 
customers. flO%) 



File^ orders. {5%) 



Work Sample Tasks 



Calls mock customers on phone, 
explains merchandise and soli cits 
purchases.. (70% J 

Records names and addresses of 
purchasers , as well as amount of 
purchase. ' (15%) . . . \ ' S 

The other 15% of the work sample 
involves non-related activities 
primarily consisting of cleaning 
the--WQ4^-^r-ea-ar Kl - rep Ta ci 



jmaterials 



One can see that the woipk sample contains approximately 85%. of 
elements which were derived from a job analysis on the job of teleph 
itor. The- obvious question that arises is what percentage of work s 
overlap is adequate for a work sample to be considered representative 
realistic worker skills? , Certainly, it is almJUft impossible to incorporate 



all of the elements of almost any job into 
tain a myrjad of task-related and behavioral 
eYi^t in the'real work environment. Thus, 
be a r00% match-up between job and work samble, elements 



orR samples, since most jobs 
ly-reiated factors which can 
will 'be very rare for there 



At the present time, there are no cleaj-cut guidelines as to vihat 
tutes an adequate matc^-up. Indeed, *all that has .real ly been determi 
that all the elements of a job need not be included in a test or work 
order for the test to be representative or valid. -Manning's (1978) i 
tation of a recent Supreme, Court decisipn indicates that, "-'.key -e'leme 
'critical skills,' or 'minimal competencies,' or 'significant compon 
a job can be used in* test validation instead of the full range of 'sl^'i 
edge, and other, attributes ^y^eated- by formal job analysis" (p. 76V. 
not all the factors inj^vecrin a job mustjbe included in a work simpl 
order for it to be rea^wably valid. However, it is recommended that 
maximum extent possible, all critical 'element;s of . a. job should be^ incVu^led 




m a work sanfp^e if 
meat potential on a 



the tool- is 
given job. 



to be usecl as a basis for detenriyining/ employ 



If it i<s- impossible to incorporate some of 



tjie critical job elements into the work sample, then perhaps other evaluation 
techn-iques should be used, i.e.,' jpb site evaluation or supplemental" work, 
samples. - .* . 

- ». . • . : . 

Insu^ring that work samples are representative .of real.istic. c'ompetitive 
porker skills is .important for maoy reasons. _ .First., if a* Kor'1< sar/iple dc5^s not 
actual ly^nteasure what it purportsi^to measu^el> th^ aec4'fe.i6nsr 'an/' "in^pi^ta- 
tions^as to employabil ity are based on a nOn-valid assessme/itftechntqu'^ which 
is potentially detrimental to a client's rehabilitation progress.'. For ex- 
ample; if a .work sample fails' to include some critical job^ tasks,- then a ^ • 
clieijt may pe able to^ perform the work sample, but when placed on the jbb*he 
finds' that 4:here are 'some important job demands which he car#iot satisfy'. This 



\ . ■• *H ■ . , , . 

' . • \ ' "» ■ , ■ - . . . 

. situation has a negative h'mpact ori' the client>, the jo^ pl*cement person,. and 
, the potential employer . It al so ^reflects poorly on the vocational evaluation 
service. Conversely, if represen^fativeness is not achieved, a client may be 

^ screened- out of a job he ican actually db simply because he" performed poorly 

. oh a work sample whjch required additional skills that -were irrelevant to job 
performance. For example, suppp'se a client is takqnq .a- SQ.lderinq ^ork saanple 
which requires him to read a complicated electronic circuitry bluepriilt even 

•. though the soldering jobs available in that geographic area do not require 
blueprint, reading, skil Is. Workers only .have to soljder together tw6 wires in 
■a stanjdardized, rotating way. In this case-, the atfditipn of a critical skilO 

•. that is not a Job requirement limits the* representati\(enes5 of !:he work sample. 
This could .easily lead' to assuming that poor work sam^ performance on the 
blueprint reading .indicates 1 imi ted -potential for empl'oyment as- a saliderer. 
Thus, the' work sample is not repres.entative of real istic local jtrk^quirements 

.Finally, one of the most basic functions of vocational evaluation is to . • 
assess cl,ie(it interest in a s pecific job or occupational area*! Therefore, in- 
~^^"of^er^lW~a^rW^pTctm s -ffxper rence- 

.with the work sample must; be based .on an accurate representation of the job. 
Otherwise, trlient interest is based on a superficial or 'incorrect understanding 
of the job. This can lead to unsatisfactory .job/trainin,q placement. For ex- 
ample, if an ^valuator wants to assess a .client's interest in the area of small 
' engin.e mechanics, the work sample .s.hould simuj:ate .the j"ob Of smal-1 engine me- 
/cJnanic as closely as possible. Thfs means that the client should work-in an . 
area-with noxious odors greasy and dirty parts,, etc. -However, in many cases, 
a client is given a shiny clean engine to work on in.a'qiuiet forma V testing 
atmosphere. This presents an inaccurate picture-of th^ work environment, 
thus, if a client expresses an interest in this. job, '-it. "may be based on an 
. inraccurate .understanding of the job. In such'a cjise^ the client's interests 
may change dramatical ly' once he is placed on the job .j Therefore, if the- 
e'valuat'ion setting doei not do a. reasonably good job of "approximating the 
real worj:. environment, the evaluat6"r should give serious consideration .t?D 
substi tutin.g» job site evaluation, techniques for. work samples since the former 
is usual ly. a 'better model of the' job environment. . 

- r X' 

In summary, the primary rationale for-CARF Standards 3. 4. 3. 1.1. 7. 2. a 
'^nd the first part of "b" is that they hel^ insure that. a work sample actually 
measures what" it purports to measure and is rep»te'sentati ve of r^al competitive 
worker skills.. Thus, the data which is derivefL from work sample' testing 
provides a meaningful arid accurate picture of a client's vocational potential. 
Jhen cTient, evaluator, and referral sou^rce have- a valid and rL^liable source 
' of information "for decision making. . 




PART II < 
« • ■ * 

g' Work Sample Standardization 

' * -. . ' * 

....... ... * . ' • ■ 

Part "b" of standard S.A.C^l. 1.7.2 states: ' " •• 

work samples shall be established by an analysis of job tasks or . 
traits related to a specifie area of work, and be ^standardized 
••' as to materials, layout, instructions, and scoring. (CARF, 1978, 
. p. 28) . ^ ,, , ■ 

This ^guideline actually has two sections. The first indicates that work 
sampli&^are to be based on an analysis of a job or group of jobs, while the 
second -suggests that work samples s-hould 'be standardized. Part I of this pub- 
lication has alrs^ady explained the importance of analyzing a job in order, to 
insure work "sampl-e representativeness. Therefore, Part II will be confined to 



a discussit)ri" of the concept of work sample standardization. . ^ 

CARF has provided specific examples of what is to* be, included in a stan- 
dardized work sample. * 

• ■ 
♦ ■ . 

Each work sample shall have. an examiner's manual' which specifies: 
"(1) its relationship to the DOT, Oecupationa.1 Divisions, Worker 
Trait Groups, or some appropriate job analysis system;* (2) pre- 
requisites, i.e., any specffic work sample task requirements 
which might make administration o^" the sample unfeasible for a- 
given individual; (3) the work sample purpose, i.e., specifi- 
cally what is the sample attempting to assess; /(4) the materials 
and equipment used; (5) preparations for testing and the layout ' 
of . materials;- (6) instructions to th*e individual ;^ (7) instruc- . ' 
tions fpj* timing, evaluating errors, and.scorihgVl£-4PPlicable; 
(8) instructions for interpreting scores. . (p. 78) ^ . 

From this, interpretation one can see' that there are some relativel^^ ex- 
plicit requirements wh,ich define what constitutes work sample .standardization. 
The following examp^les provide' a 'further clarification and, interpretation for 
this guideline and readers are referrred to' the Materials Development Center • 
(MDC) publication Wo rk- Samp le Ma nual Format for further information. 

H. Job, Analysis Relationship • , ^ 

« 

The job analysis relationship should be based on a standardized job in- 
formation system which provides_organized daf'a as to the 'jobs, traits, work 
activities, etc<7\di recti y related to the work sample. The information m'ight 
be presjMtedyiTi tlye following manner. . ' 




sample wate based on an analysis (J'^vjhe job of Watch Re- .« 
)a1 rer (clopiiland-watch) , Dictionary of Occyp^tional- Titles (DOT), 
mrth Ecji^on, Code 715.281-010. In the DOT classification system, 
^a-fiata-l^p4€-TM^'i9s-4DP-T^--4;o4e-of-^^ 



requires an "analyzing" relationship to Data,Ta "taking instructions- 
helping" relationship to People, and a "precision working" relation- 



ship to Things (pp. 1369-1371). 




According to the Worker Tra i t. Group Gui de (1978) of the 'AppaTachia 
Educational Laboratory (AEL) system, the job of watch repairer falls 
in the Career Area of Industrial-, Code 06., and t^e Worker Trait- 
Group (WTG) of Production Technology, WTG 06.01 (pp. 189-195). Work 
activities include: 



' 1. Activities dealing with things and objects .... 
9. Activities involving process, methods, or machines , 
l(jl Activities involving working on or^ producing things, (p. 193) 

In this example, two different job description systems have been used, the 
DOT and tjie AEL.. Both of these systems provide concrete and specffic informa- 
tion as to job related factors. Additional data such as physical demands, 
aptitudes, interests* related jobs, etc., could also be included in this 
section. . * » ■ 

' ■ . ^ ; ' 

2. Prerequisites ^ * ^ ^ 

-- ...... __ _ 

MDC_(1977) defined work sample prerequisites as ''"mini mum performance re- 
quirements from prewi pus- tests* or work sample? that will nee.d to be met before 
the work sample'jnay 5e administered" (p. 4). Examples of prerequisites. might 
include: reading, levels, physical and medical factors, specific aptitudes, 
educational training, etc. All prerequisites should be listed, ^.g., be able 
to lift and^arry 50 pound boxes, and any precautions such as safety hazards. 

3. Work Sample Purpose I 

« 

y It is important to clearly identify in the work sample manual what specific 
factors the work sample is designed to assess. These factors should be as 
clearly stated as possible. General statements such as, the purpose of this 
work sample is to assess "production skills," "clerical skills," "potential for 
compeytive employment," etc. , are too vague in that they don't indicate the 
exact^^skills, ^titudes, traits, etc., being assessed. Look at the following 
example: 



* ■ f 

Assessment Description - The purpose of this work sample is to assess 
the. following, factors: 
• '1 - ^ * ■ ■ 

Ability to wprk-^^competitively as a drill -pVess operator (wood). 
Ability to use rulers, micrometers, andcalipers to measure wood 
■ stodks* ' 

3. Ability to read blueprints as* they relate to drill press operators. 
4.. Safety habits with regard to operating a drill -press. V 

5. Interest in' working on routine, repetitive machine (drill press) 
operating controlling tffsks. 

6. Ability to stand for periods ranging up to three consecutive hours. 

7. Spatial, form perception, motor coordination, finger dexterity, and 
manual dexterity as defined and rated by the Dictionary of Occupa - 
tional -Titles , Third Edition, Volume II, page 663. 

This type of a ssesSsa^nt description gives the evaluator as well as the 
cTierrr a'^Tear-Cut'Qffife^ specific purposes-t)f the-i^ork sampl-e, 

If a work sample assesses specific traits or oth^r factors, then they might 
be listed in a similar fashion, e.g., spatial aptitudes, manual dexterity, - 
GEO, Data-People-Things codes, etc. 



All af the materials- and equipment necessary in order to use a work sample 
shoultf bp listed in the manual . "Information inc)uding%>art names, quantities, 
purchase opder numbers, shippwig weights, sizes and descriptions such as color, 
••^rade, etc., purchase source and address, and total quantity price are neces- 
^r.Y'!:..(MPC, 1977, p. 2.6)j this information helps injure thit lost or worr^out' 



parts and materials are replaced in a consistent manner so that the corrstruc- 
tion of the work sample does not change. .If .it .should dianqe,*it may mean that 
norms, reliability, validiitj^, job analysi^'.information, etc., mSy. no; longer be 
applicable, -itr • 1 



5. Test Preparation and Layout 



T-est preparation and l-aybut.tasii^^^^^ to information, related to, 

work sample conditions, administra^t^rt^equipment, and setup and breakdown 
■procedures. ■.■ ■■' a;'-:-— - . -^f* 



The test environment should be xiescri bed, e.g., indoors vs» outdoors, 
quiet* vs^ noisy, arrd each vJor4< s#mpl^e^4^ li'st the equipment that is 
necessary in order to proceedwith the administration. Examples include:- 
tools, hardware, safety devices, tables, manual .s, scorin.g/sheets, timers, eitc.' 
In addition to this, every work sampl a must b^ set up in^a certain manner. ■ 
The locatiorf.of the tools, their arran4pnient,\i the method .o,f setting up the ' " 
work station, etc., shouTd al 1 be clearly spelled out. This will iielp to 
insure standardized administration procedures,. . If norms or industrial Stan-- 
dards are u.sed, it is particularly important to make sure that 'th'S work sample 
layout is the same as the layout us$d during- the nocming prociess'i;!^^^^ means 
havinn objects positioned in well-defined locations with dis|aneef japl Ted but-. 
In most cases, a. diagram ef the work sample Hayout is yery^ttpfuli 




Work Sample Layout. 
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6. Instructions 



V 



'Instructions to the ;[ndividual refers t^that information which is neces- 
sary for administration."" It will often include sJome of . the previously dis- 
cussed items such as prerequisites; materials and equipment, and preparations 
for testing and layout. Instructions are generally provided fer both' t he- 
el ienrt and evaluat(<^. The instructipna-l procedures should be ^as specific a§ 
possible and should be standardized wi:^h regard JU^t he format in which they 
are initially presentecj, e.g;, -written," oral , demo^^ratton. and haoxls on.. 



« • 



If i client is unable to fallow the..S*taridardfzed instructional format, it 
should be noted that other formats can be pnovi-ded, but they need not be 
included in the work sample manual; for further discussion 5f this *stJbject 
f^fer to McCray. An example of instructioris for both the client -and 

eva>u^pr are. noted below: . • . . ' , 



ft ' 



> ' .CI i ept I n s t r u c t i 0 n s (Jo be. r,ead to the client by the evaluator) - 

"Your jo^H^^-to take- eaeh Of the 'bolts in this 6ox (point) and ' ; 

• ^ thread. a niit- from tl^is.^box (point) onto the bolt, and put the . 
finished assembly Into^tl^S box (point)." ' * 

• ' .: ♦ : . ■» - ■■, • ; .■ ■■ . ' • . 

EvaHiatpr -Instructions' As the gvaluator points to -each- of the ; ■. - . 

*• ^ firs.t tvvfo boxe«, he is to remove- one sample object and-shovf it to 

the client- ' At- t'hS^'same. ttme •he--wil 1 assemble ^o*ie nut' and bolt; .\ 

. and place it in" the box loteled for fin«ished assemblies. : ' 

• * ' . • . • tf. ' 

' . ■" - - . ^ < ' ' ^ 

Thus,, it is -evident that specific directions are to be provided; howfever, 

this exampl^e is by no, means 'complete.' If s pa ce\ permitted, it would go on ,to 

^meticulously explain the qx^ct production method,' p.ractice period, objectives,' 

performance session requirements, etc. • * 

7. Time, Error, and Scoring Instructions .. ' * " 

This section refers to the basic scoring criteria which are used for 
analyzing a clien^t's performance. The techniques to be used in -scoring, 
quantity, quality , errors , '^'etc. , must be clearly .identified along with all 
' other measurab 1 e^Vutcbniesr. ' In'.-addit10ri "to-?thls;^nf(XriTiatiW§a'ch'aT~d6mf^^^^^^^ '^""^ 
tions a's to what constitutes an error, when timing should begin and end, 
behavior observation requirements, definitions of relevant behaviors*' etc. , ^ 
should also be included. 



8.* Instructions for Interpreting Scores 
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•• This information should provide special input into understandtng'a client's 
performance, ft may inclu(;Je instructions as to how-to read tables ' in order to 
identify appropriate norm groups. Other information might" include how the 
norms were developed, whether they Ripply to experienced or non-experienced • 
workers, learning curves, situational factors suchas behavior probTems, etc. 

Work sample standardization is important for many reasons. First it helps 
insure that. a client's performance is not unduly influenced by haphazard 
assessment procedures which may contribute to unreliable results, Second, it ■ 
provides an objective basis for making consistent and valid observations be-, 
cause this skills, behaviors, concepts, etc., which the j^jork sample is designed 
.to assess-are clearly defined and identified. Thus the evaluator and the 
client areboth aware of the specific reason(s) for using the work sample." 
This acfds meaningful ness to the technique and helps reduce, the likelihood of 
individuals misunderstanding the purpose(s) of the test. Last, a standardized 
manual .can be very iise/ul for training new staff members. With a manual, 
should an evaluator leave a facility, he does not take with him all the knowl- 
edge and insight necessary in order to use the instrument effectively. This 
means t^at a standardized manual can help prevent "reinventing the.wheel" or 
having to organize and *rfevel op the same work sample over and over. ^A11 the«se 
•benefits point out' that work sample standardization is a critical aspect of 
an evaluation pr.ogram_ since it facilitates a systematic analys^is of client 

* . « " 

; / ' ■ 13 - 




•capabilities. Thus all clients are essentially given an equal chance to 
achieve maximOm 'performance.' ' ' • . 
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• ■ PART. Ill* . ■ . - ' 

" * 

\-ioY± Sample }^ms ; ' . . 

An ind(^ciual ' s perfcft^mance score on a- work sample is most us^'fuT when * * 
other scoress, are available folr comparison',, '.The choice of a. comparison score 
*or group.^ofi cores shouTd be guided by the underlying purpose for v^ich^a work 
sample i^TBeipg used. Since 'Work spples are common.ly- used as an aid in the 
assessment of a client's potential for' functioning in various competitive * 
* bccupations, it seems logical that work "sample' comparison scores based on 
performance rates of- workers from competitive *jndus'ti!;y should be used v7henever 
possi'ble. Preferably, these 'competitive-standards wiVT reflect the pfoduction 
expectations for newly employed workers rather than experienced workers who in " r- 
most tases perform at higher levels than inexperfenced' employees. 

Part "c" of standard 3.4.3.1. 1.7.2 slates that when using work samples: x 
"competiti\/e norms or industrial 'standards shall be established and used" 
few,.. 1978, p. -28) . . The CARF-VEWAA interpretation of thi-s guideline further 
explains: ' • ' 

There. should be evidence (such as time studies; Methods-Time-Measure- 
ment (MTM), employer opinions as to relevance of work sample^content 
or quality and quantity standards, actual competitive worker perfor- 
. mance data, etc.) that performance standards used for interpretiv'e - . 

purposes have" been rela^ted to app^opriate competitive performance 
standards. (Vocational Evaluation and Work Adjustment Association 

anH rnnmi<z^Ah'n nn ArrrPfiitatinn nf Rpl^ahn.i t.a tinn Facilities. 1978. 

• p. 6) ' , _ , 

The standard and the interpretation stress that all work samples used for 
assessing client potential 'for competitive occMpations should have competitive 
norms or industrial^standards, available. In this case the terms norms and 
industrial stand'ards both ref^r to the concept of an average' (me'an or median) 

'.performance score for a ta^ or series of tasks as well as the range of scores 
deviating above and below the average. Competitive norms and industrial stan- 
dards are extremely advantageous because they offer a comparison group for ^ 

"determining not only whether or not a client can perforf\i a job, 'but how well 
the client can do the job. For example, if a client is„giv,en a S^imple .nut and , 
bolt assembly work saimple without norms, the evaluator can on ly- (determine ^ . 

whether br no,t the client can actually correctly perform the assemblies, e.g., 
does he have adequate dexterity or can he remember the sequence of operations. 
The evaluator only has a limited idea of how well the client can do the job. 
The question as to whether or not he can perform the assemblies as' fast as 
employed nut and bolt Wsemblers has still not been answered. This question 
can, however, be an^weredvf the evaluator knows what t^e competitive pro- 
duction standards are» Por example, if newly hired workers (e.g., less than 
40 hours nut and bolt assembly experience) are expected to perform a minimum 
of 100 assemblies per hour, then the ,evali3ator hds a relatively easy job of 
comparing the client's performance to the competitive standards and thereby 
miaiking more reliable decisions as to employment potential. 



*Much of the material provided in Part III of this publication was taken with 
permission from »the authors, C. Thomas Allen and Arnold §ax, of an earlier MDC 
publication entitled Norms and Performance Standards for Work Sample Scores . 



Perhaps the most important point of- this guide! irte 'is that competftive 
norms or industria.1 standards 'musi be used.. The term competi»tive denotes the 
concept that the norms or i'ndustrial standards should be based oh people ac-tu- - 
ally employed in occupatioiTi, requiring the skills, traits-, etc., requVei^ of > 
the work samp'le rather than a handicapped population or general group of people* 
for which there is no objective evidence as to wjiether .or not they could really " - . . 
do the jofcf(s) . 'In addition to this, the, industrial or compe-titiVe' star\dard {, 
used in a work Sample should ■reflect the^starjdards used in industry for the .* * . 
same job.' For example", an MTM study, on. a work sample. might injJicate that 300 ^ 
pieces per hSur-is 100% competitive, and it can easily be erroneffOfily assumed 
thaf this is the 'standard industry uses for worker perfoniiance-. -Yet, .in actual 
practice, an industry might regularly employ people who perform^ at on.l*y 70X of » ' ' 
the work sample's industrial standard. ThiTs thg industrial standard used in , 
the'work sample shoiild, a^^s much as possible, reflect the actiial practices use.d 
''in industry. ' . • / . * " 

%. ■ • • - " ■ . 

• • • • . ' , _ 

Norms and Industrial Standards, * • . . 

Norms are needed In order, to compare an individual '5 performance to other 
appropriate reference groups. With regard to vocational evaluation, such ref- 
erence groups shauld generally be composed of workers employed in occupations . 
closely related to the work sample. The usefulness of a nornj^oup is deter- 
mined to a great extent by the method's used to select its members. Care must' 
be 'taken to insure that members oT the norm group are representative of popula- 
tions to which comparisoiis are to be made. For example, a work sample that , 

flisspssp^ .a.n individual's potential to work as a small engin% mechanic would '■ 

more appropriately have a norm group of successful ly 'employed small engtne 
mechanics or some other closely related occupational group, rather than^a norm 
group of. college students. Regardless, of the degree of planning, that goes into 
the development of norms, one must collect data "on individuals that are*avail-. 
able -for testing. As a minimum requirement, one should provide an accurate 
description of members of the norm group. There are three different types of , . 
norm. groups which generally apply to work sample testing: 

1. Client norm g>;pups * • . / " # 

2. General population norm groups 

3. Competitive norm groups ar industrial standards 

1. Client Norm Groups . » - , . r'. 

■ < » ■ 

-. In many work evaluation settings tljie most commonly used norms are those. 

based on scores of otFfer clients who have performed a ^specific work sample. 

These are not, however, competitive in nature and, therefore, do not satisfy 

CARF requirements.^ .h* . 

Even though client norms may be relativeTy easily obtained,- -there are many 
serious problems with using them. 'Client norms are dependient upon the extent 
of abilities (or disabilities) of the clients served by a particular facility. 
Since client groups may vary considerably among particular facilities, norms 
based on clients' from one facility may be misleading for use in another 
facilHy without detailed descriptions of clients inclucled in"the norm group. 
Also-, within a par^'cuUr faci'lity norm groups may differ for each work sample. 
One work sample may be given to almost every client.. Another may bh adminis- 
tered only to .those that an evaluator has judged from^rior testing to function" 



> 
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at farily high leveYs. Thus, a. client whose performance ranks relatively high . 

■ ■ ' ■ as compared 
^ame skills 
a client's. \ ' ) • 

performance ma^ lippear to be above 'average^ when compared to other' client^* ' \ 
Yet, if t"he client's'perfcrman<^>/as compared to ^competitively employed workers - • 
who' represent the group the -cl 6rkj will , actually have to compete with fcyr em.- . ' 
Hoymenl, his performance level might be coasiderably .below competitive, . . , 
"stand-ards. Thus", using client norms can easily lead to 'the erroneous cfcn-^ 
Glusion'that a- cliejit has potential in a work area, which ynay. actua-lly be 
j'oappropria-te 'for him. Becai*se of this problem,, it j.s iirip^rati've that if. an ^ . 
**'evaluator is consi-deririg the clJent forr competitive 'pi ac9men.1i» cOmpetitiv^e 
norms must be .used since, client nbrMs will not specify whf3<* performance -^"^ 
dards are necessary for successf ul employmelit. ' V" , r ' 

" .• " . ■■■^ " . ' M'-- ■■ 

In spite of the limitations i a using client noiifis, there is one instance. 
• where they can be very useful. That is wbW a client is being evaluated with ' 
regard' to placing him in sheltered employment. In, this, case., the worker popu- 
lation which the client will be competing with is actually a client group, 
rather than competitive workers employed by industry. However, the»workshop \ 

• in which the client is to be placed must'sti.ll have .competitive production k 
standards available for meas\jring the.ir client.'s performance relative to the ^ . 
industrial standard. Thus; an evaluator might know that a sheltered employment 
program is involved in an assenjbly operatign whose competitive industrial pro- 
duction standard is 100 pieces >er hour. Worksho'p production rrecords indicate 
that the average productivity , Of their clients, however, is only 4a pieces per 

— hour or ^0?^ of the industrial stahdar|. — S o if th^ evaUiatnr h a s , a client whnse 
performance is 60 pieces per hour on the. work_sample which- was derived from the 
workshop task, the -evaluator should recognize two important things. The client 

• is produciilig below competitive industrial standards, but if placed in the 
workshop his production skills, are compatible with the other sheltered em-^ 
ployees. : In this ca"se, clien^t norms would be useful for determining a new 
clieVit's ability to produce at levels Comparable to the standards required of 

. a specific sheltered workshop.- • ' . 

2. General Population- Norm Groups * 

The secorjd type of norm group is based on a general population of non- 
disabled subjects. The evaluator. administers a work sample to a, random sample 
'of non-disabled, people who are not competitively employed in jobs related to 
the work sample. This group is-referred to as a samp-le of the^ general popu- 

> latibn. In most c^ses, however, it is impractical for. Valuators to obtain a 
true randdrri sample of the general population. Instead, the non-disabled 
population is made up of a select group of individuals who a^e readily avail- 
able, .e.g. workshop staff, students, or volunteers'. Like the use of client 
norms', thHs method of comparing- client performance has seri,ous limitations. . 
Work sample .norms based oh the performance of -members of a general , nori- 
disabVed population offer no direct ba'sis for>^ determining whewier a client 
i§ capable of functioning at or near the rate of competitive workers on a-- 
parti.cuTar work' tiisk, since none of these people are employed in a job directly 

-•related, to the work, sample, and it cannot be assumed that they would be sue- . 
Cessful worker's simply because they are not disabled. Thus, predictions as to 
client performance capability are unsubstantiated and unreliable if they are 
based on either client or genera-l population norms. ' * 



3m Competitive Norm Groups or Industria> Standards 
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previously, whenever* an evaluator'^^eelts a direct basis for * * 
determining potential for success in competitive situations, competitive norms ~ 
or Industrial standards based on the perjormance rates of worijers in competi- ^ 
tlve indust»*y should be used. These are the only techniques which satisfy '-^ . 
CARF guidelines. This is because they provide a reliable 'and di feet bas^is.for, ".^^^ 
.comparison of" cl ient^ performance 'versus real work r^quiremetits. • JV** 

Competitive norms 3re' more difficult to^oStain than noncompetitive nc^Tiis; . . 
however^ the benefits justify the effort-. One method of obtaining competitive. • 
norms Involves having workers empToyed in -jjobs clos'^ly delated to the work 
• sample actually 'perform the work sanjple.* The .results 'friay, then be statistically " 
. aiialyzed to obtain a^, relatively good picture-of competitive' performance re,- ^ 
quirements. For example, ii group of 'competitively employed $leCtron.ics 
assemblers would take an electronics assembly work sample which may not be ' 
exactly the same as their job but which involves the basic critical elements 
of the job. Based on the ' perf ojrmanoe of this group, the evaluator. should have 
a.' sound idea of how well competitively employed worlRers can perform on that * 
particular task. This technique is particularly useful* when an evaluator has 
a work' sample which ha,s already been developed, but lacks any normative data. 
It can also be useful for a work sample which is based on a job taken from 
Industry, whicli has no concrete production standards. For example, a service 
occupation such as maid service may not have any specific .production standards, ^ 
but based on a job analysis, theVeval uator develops a maid "service work sample.- — - 
In order to obtain norms for such a work sampla^ the evaluatof may choose to' • 

hav e s ev eral pa o p la wh o are c ompetitively employed as maids perform the work — 

sample. Then the appropriate data such as speed, quality,- etc.,_ could be 

obtained ^nd analyzed so as to provide perfprmance criteria. - ' • 

^ A* second method of obtaining competi ti ye norms or I'naustrial standards 

involves developing a job sample. The job sample is probably the most ef- 
fective and efficient way to obtain an industrial standard, because 1t^is 
based on real work being done -in industry. In this case, the evaluator*. con- ^ 
tacts,. businesses and Identifies jobs that hold potential f6r the client 
population being served. A job analysis .,1s performed, and then the evaluator 
\ can create a job sample which replicates the real, job in its entirety. Once *' 
the job is accurately simulated, the evaluator may then use the same production 
standards, being used by the industry from which the job was taken. In. effect; ■ 
the evaluator simply borrows a competitive production "Standard which industry 
has taken the time and. trouble to develop. This approach can save the evalua- * 
tpr a great deal of time and effort si nc"?* the problems' of soliciting volunteer 
.' workers from industry, accumulating data, and synthesizing/analyzing results 
are eliminated. The second advantage is that rapport is established with 
potential client employers and specific jobs are pinpointed. The evaluation 
. unit may, therefore, %^rye as -a more effective placement tool. Third, the 
job sample inherently has a great deal of validity since it fully replicates 
a job, with the possible exception of matching fhe entire indlistrial environ- 
'• 'ment. Fourth, industry should be able to speciTy such factors that might be 
observed in the first few days which may distinguish between those new em- 
ployees that eventually become good producers from those who are marginal or 
unacceptable: in production.' ^It should be emphasized,' however, that it is 
imperative that if production standards are taken from industry, the job '• 
sampTe'must replicate j:he job in every way.- The same. production methods, 
layouts, tools, equipment, etc., must be used for both the job sample and 
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the*jobH Otherwise, the standarc^s^may nor loager be vali,d since different 

procedures may strongly influence performance (for a^ further discussion of 

the job -sample, refer to Pi lien, 1977). V. ' - ^ . - . 

.-"The third method of* obtaining' competitive pei>formance standards foy> work. . 
samples involves establishing- production standards which directly apply to 
the work sample. If an. evaluator h^s a work^mpl? which,' needs to be normed, 
but neither o€ the first two techniques are. p(^sible, e.g., (1) there ar.e too • 
few competitively employed workers available W taking the work' sample or the 
jobs are toa-4i-sstmflar., or C'2) the pi^oduc^i^n. standards' us.ed by industry ^fan't 
be applied because a jo^-sample couldp't be developed or the industj^y had, no 
specific production standards,: then pr6ducti^ standards may still.- be estab"- v. 
lished By^sing work measurement 'techniques borrowed frpm. industry tthe 
different kinds of wor|i.measuif'eme,nt* techniques will be di^cussedvin the^fol-, 
lowing section).^ Rather than bringing workers in tp^.take the work sample, 
or developing job s^mplesl this procedure requires .that the evaluator contact 
a' qualified engineer ;4o perform time\ studies, Methods^-Time-Me'asurement, 
Master-Standard-D^'ta, or'other work measurement techniques to -arrive -at re- 
liable, objective production standards for the work sample, (re^er to Botterbusch, 
1975, for further-discussion).^ Although this procedure is useful, it also has 
a significant limitation in that it may be very costly to have an engineer comet 
in and analyze a w6rk sample and develop .production standards. This is partic-/ 
ularly true in cases where many work samples are involved or individual work 
samples a.re very lengthy operations of several minutes or even hours. It may 
take several hours just to analyze a two or three niinute job, and thi^ problem . 
must be taken into consideration when selecting a method for obtaining per- • 
fnrmanre 'Standards. Qne way to minimize this expense is to try and objUir),.^., 
qual.ified volunteers through cortimunity invpTvement or Dy ^bntactiny dp^m)pridt■e 
professional organizations. Interested parties may wish to get in touch with 

,thei • , ' \ 

' ' . American Institute of Industrial Engineers, ms^t 

25 Techno'logy Park/Atlanta ' • • 

Nforcross, "Georgia 30092 • . . 

■ * * ' 

They ^ay tfbl e to provide further informatio/i with regard to locating the 
nearest chapter. • . 

f 

i ■'■ *WorJ< Measurement Applied to Work Samples 

■ .. - . * ' * ' ' 

Industrial sitandards are" usual ly derived from a variety of work measure- 
ment techniques. Work measurement techniques have been developed by' industr^ial 
engineer? and used successfully to estimate labor costs for future projects, 
schedule production, evaluate different methods ofv perf ormi ng a job, andf^es- 
jta^lish incentive wages* Work -measurement involves the detailed Analysis of a 
•particular task using standardized techniques in order to determirte- the amount 
Of time needed for an average or normal experienced worker to perform the task 
under normal conditions. The industrial engineer is concerned not only with 
establishing an accMrate estimate of the time needed to perform a task, but 
also in evaluating the most effective methol^ used to perform the task. Meth- 
ods of setting up the work task are examiti^ to eliminate such factors as 
awkward motions*, long reaches, and excessive material handling (for a film 
review of \tjiis subject, refer to Botterbusch, 1975). 



*. The two most 'popular ^yst^ms. of work measUremeint are* th.e time study'and 
predetermined motion time systems. Each type offers specific 'advantages de-»**^ 
pending on a variety of circumstances and the types^.^f work tasks to be studied. 
Time study is 'a method of determining how much time a task sh8uld requvre.. A 
cToci<', stopwatch, -or other timing device. is used, and the time , that elapse^s 
while the task is being done is recorded. Usually a series o*f these observa- 
tions or timings ta.ke place using a group of aver'age , experienced workers; An 
average tilji.e is then determined from the* performeince of individuals within the 
reference g/oup, and this time is •thougfrt to/fepresenj^he normal amount of 



time required for an average ^xperience'd-s^j/prk^r tOvjJ^rform the desired opera-^ 

'^^tio.n(s). A predetermined time system is jan industrial enflineerinq technique 

• tfjat allows pr-oduction! time standards to determined wi-thout .relying on' an 

average, experienced worker. Instead. "of timing the mo.t.ions of . individual 

workers, th-e speeif>ic moti£)ns requi^ed-'to do tih'e^job are ideniti\fied' aqd listed. 

^on an analysis sheet. A time is^then assigned**t6 each of. these movements. 

These ti-mes are taken from standa'rd.ized tables whose values .have been pre- "* 

detet»mine£f, i.e., thousands of manual motions were observed and broken down 

into a few well-defined motions whose performance tim^ was Statistically 

analyzed and averaged.. .The theory behind these systems is that elements of 

human movement', .on the average , take the same amount of time to perfojjpi, .re- 

gar'dlessyof the individual worker involved, these systems are quite reliable 

•and are widely used by government' and indus'try. For fyrther discussion of. 

this subject, refer- to Botterbusch " (1975).. • . 

* . . ' • . ' ' > ■. • . 

Because time study requires the observation. Qf average, experienced 
workqi^s which are generally not available in most workshop or evaluation set- 
tings # predetermined motion time systems are often more appropriate for 
developing per formance standards for work samples. The exception ^to this is 
•the Ccise where the evaluator can bring "in average, experienced wprke^rs^'Tronr^^ — ~ 
industry or workshops and use them as the basis for the time studies. Howr 
ever, when average, experienced ^workers are ava.il able, corresponding pro- 
duction standards will usually 'already have been d«velopecl since the job is 
already In progress. Thus, the evaluator may be.able^to save'trme and money 
by simply jDdrrowing the standards already^ developed by industry. Of the several 
standardized predetermined, motion time systems ,'Metliods-Tfme-Measurement (MTM) 
appears to be the most widely known and. commonly used. . Some other system? 
•include Modular Arrangement of Predetermined Time Standards (MODAPTS) and 
Master-^Standard-Data. For more information the reader should contact the 
nearest industrial engineering organization. 

Work measurement may &e used .to provide valid and re'alistic performance 
.stanclards for man-y^worlc samples , -especial ly those involving repetitive manual 
'.operations. The work sample need not be a work task performed in'industry. 
Work measurement- may b& used regardless of whether a work sample is ^a simula- 
tion of a specific Industrial job or a task which is Intended to relate to a 
variety of jobs. If the work Sample task were to. become ffn actual job in- 
Industry, the work -measurement results should be a good estimate^of the per- 
formance rate of an* "average" worker. S'ince there is no imnediate need to 
gather normative data, new wprk samples for. which work measurement standards 
have been developed are immediately usable for evaluation purposes. 

A "client does not have to meet or exceed the industrial performance 
standard in order to be considered to'^ve a good chance for success in per-^ 
forming task? required in jobs related to. a v^ork sample task. .Performance 
.equal to the standard is' designated 100% performance.' If- the task is a type 
'In which, the establishment of work rhythm is 'important, performance at 30% to 
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SQX of standard might be considered "^HSlfifod on the first trial of the work 
sample. Due to varying levels of p.hy?!^^patigue and emotional stress in. the 
eyaluati on situation, a cjient's work perfofw»ce may differ cQnsiderai)ly from 
.time .to time. It is generally preferable tha^a client be "administered thfe 
work sample several times. Since a work measurement standard is based upon 
the performance of the Average experienc'ed worker, repeated trials of a work . 
satnple aie gehe^ally advised. Repeated trials allow opportunity to observe 
signs of interest in the work tasks. Close supervision is" generally not re- 
quired after the client has >1 ear/fed to perform the task correctly. It v^ould 
be daf\gerous to specify a peycfentage of an~ average compatitive rate that might 
be used as an absolute "lower "limit for judging that a client has potential for 
''\;ffyctessfuVcmp>oyinpnt. Any "guidelines us^d must be tempered by consideration 
of variables such* as task complexity., amount of practice, and client anxiety. 
Beginning and ma»*ginal workers generally perform at rates of at least 70% of 
standaprd. Although^ this 70% figure might be used as a rough guideline, it , 
should not be considered a' firm cutoff ppint. An industrial engineer who 
develpps a performance standard should be able to provide general gyidelines, 
in light of complexity of the lask and local industrial expectatiojps. 
•» ■ . ■ ' * . • ■ 

. * Cautions in Using Competiti^ Norms or Industrial Standards 

When using competitive norms or industrial standards for interpretive 
purposes, there are certain cautions which must be observed. First, if an 
evaluator a$lminister*s a work sample to a competitively employed group of 
workers, he must be certain that the work sample is very closely related to 
the job performed by members of the f eference group, i.e., an electronics • ' 

s o l d e H f» g - work - sampl - e -should bo administor o d t o alafironics «;filflprpr*i flnri nnt 

a'group of heliare welders. Otherwise, the evaluator might' find that he has 
competitive norms but ^ey are essentially useless for predicting job poten- 
' tial. For example, if an evaluator has a cashiering -work sample* he s*hould not 
select .several clerk typists for the norm group. Such a group woulfJ be in- 
appropriate since no matter where a'^client^'s performance was in comparison to 
the competitively empToyed norm group, t hare would be no evidence to suggest 
' that the client could^or could not work competitively as a cashier since there 
is no evidence that the clerk typists would be successful cashiers. In such a 
V case, all that could be said is that in comparison to a group of clerk typists, 

• the client's perfor^iance score fell at a certain level. Although, in this 
' case, competitive norm^ have been used, the information derived .from these 

normk is useless and could be' misleading. . > 




jecond, when work measurement is involved, one must not only be concerned 
with the time needed to perfprm a t^sk, but 'also the methods used to perform 
the task. In order for work measurement standards to .remain valid, it is im- 
perative that the/mel^od used while establishing the industrial standard be 
precisely duplicated by the work sample method. This means all the motions, 
distances,- tools, etcl , must be exactly the same. If the method is changed, - 
the previous standard must be replaced by an updated standard -based oh the. new 
method. Regardless of Jhe type ^f instructions used'for the work sample, the 
client must demonstrate his competency in correctly performing all components 
of, the task, in the .standardized manner before his speed of production is as- 
sessed (McCray,' 1979)^ Instructions or demonstrations should be repeated or 
modified if necessary tG insure that the client performs the task correctly. 
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'Third, once a competitive standard is obtained for a work, sample, 
evaluator must not be cotl^ent with accepting it alone as a rigi'd, inflexible 
/ criteria for predi-cting^-successfu-l employment. Instead, the evaluator ifiust 
bg. aware of what the prevailing production standards are for competitively^ ^ 
employed workers in similar jobs in the surrounding community, since the actual 
standards used by industry may be different than the Standards for the wprk 
samiile. Fori^xample, although an MTM study might indicate that 10Q%<competi- 
tive on an asSemWy work, sample is defined ^ 50 pieces -per hour, ^industry may 
bd willing to employ workers whQ are fubstanti ally less thari 100% 'productive. 
■ Successful employees may. be producing at only 70% of the standard or 35 pieces 
*per hour. Or a business may hire new employees who are only 50% productive 
• during the first few weeks of training, Thus^ as much as pps'sible, the eva.lr' 
uator-must be aware of the r.eal performance requirements of Industry if a 
.reliable prediction as to^^mpToymeht potential is to be obtained. ^ 

Tinally, a major area of concern in using industrial standards is that 
these' standards are based on average , experienced workers. In some cases this 
, may mean that the ^iverage competitively en1p.l6yed worker is allowed hundreds or 
even thousands of practice trials before they are expected to achieve a. mini-* 
mum performance standard. Thus, comparing an inexperienced cl ient's. perfor- 
mance to the standards of experienced workers, and. using this as the basis for 
predicting vocational potential can be misleading. Under these conditions, 
one can only be relatively certsKn of an individual's performance potential 
when he scores .high in comparison to the industrial standard. For example,.^ 
if on the first few trials a client achieves 80% of the industrial standard' 
expected of an experienced worker, the evaluator can be reasonably sure that 
the client has the skills and aptitudes required to do the job. Yet, if a 

cl ient only scores 50% of the industrial standardj the evaluator cannot be 

certain that the cl ient does" not" have The potentTaT Td'^oThe job, artd "i 
unrealistic to provide the client with an indefinite number of additional * 
trials. Unless the evaluator knows how many trials it typically takes to 
achieve competitive, performance, the evaluator cannot assume that a. client's 
potential in a' given area is limited. For example, even though 50% of the 
industrial standard jnay seem low, a closer look at the standard might indicate 
that it usually takes 100 trials for inexperienced workers, to achieve this 
level of pr(\f iciency. Perhaps the best way: to resolve this dilemma t« tc 

^ obtain local industrial" standards for inexperienced or new en)pioyees as welT 
as experienced workers. Thus, a local industry may regularly Fiire and empl-oy 
'inexperienced workers whose production standard during the first 20,0 Ijburs of 
work is only 50% of ^the experienced workers' standards. Such an -approach 
allows the evaluator to compare an ,in'experierjT:ed worker's performance with 

.standards for inexperienced- as well as average, experienced workers. 

When used by adequately trained and experienced industrial and' vocational 
specialists, work measurement techniques are both reliable and valid. Although 
errors of judgment can and do occur on -occasion, well trained and experienced 
work measuy^ement specialists are. generally able to estimate production stan- 
4^rds with error rates of 5% or less. Labor unions hc^ve generally accepted 
work measurement performance standards", when agreed safeguards -are provided 
to assure accuracy and there are opportunities to request reexamination of jobs 
f^!ip^^4«Jljdi standards appear to be in error. ' Thus, it is evident that work 
measurem?rrt-- techniques can provide relatively objective and accurate methods 
for determining competitive work- sample performance standajras. 
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SUMMARY 




-A work sample can be an invaluable ass'et;*lp assessing the vocational , 
skill s,':^-nterests, aptitudes, befiaviors^'ete?.vv^J^^dicap However 
this is only true when it is coniTructed in sticH^'-A^fe^e^^ "to be a representa- 
tive, systematic, and reliab-le simulation of real worW^VgWAA .and CARF, by 



establishing some concrete guidelines directly related t'|.^ork sample testing, 
have taken a 'step toward assuring that, these ^es sent ral c|?racteri sties are 
i*ft€orp.orated tn .alV moric samples. 

When evaluating the quality and^ul^fl ity of an 4ri'div^dull: wprk sample or 
£l work sSmple battery, evalua toy's should examin> them witFi a critical eye. 
Poorly developed work samples can have a, prof oundl^^: negative impact on the 

effectiveness, of a vocational evaluation2program..,.4f: ^^^^'^^.J®^^** ^ 
simple should: 



(a) 
(b) 
(c) 
(d) 




be representative of realistic competitive workeit^ills. 
be based on' job analysis techniques. ' \ ' 

be standardized . . ■ 

have -appropriate competitive or industrial norms. • 



If a work' sample is' able to meet the aforementioned criteria, it may be 
regar'ded as an important asset in assessing vocational potpntial. Thus, eval- 
ga tors, clients, and referral sources will have one more. rel iabTe technique 
Tor gaining an accurate and meaningful understanding of client capabilities. 
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